Automatic Facial Action Analysis

This thesis provides a fully automatic framework to analyze the facial actions and head gestures in real time. This framework can be used in scenarios where the machine needs a perceptual ability to recognize, model and analyze the facial actions and head gestures in real time without any manual intervention. Rather than trying to recognize speci c prototypical emotional expressions like joy, anger, surprise and fear, this system aims to recognize the head gestures and the upper facial action units such as eyebrow raises, frowns and squints. These facial action units (AUs) are enumerated in Paul Ekman's Facial Action Coding System (FACS) [17] and are essentially building blocks, which can be assembled to form facial expressions. The system rst robustly tracks the pupils using an infrared sensitive camera equipped with infrared LEDs. For each frame, the pupil positions are used to localize regions of eyes and eyebrow, which are analyzed using statistical techniques to recover parameters that relate to the shape of the facial features. These parameters are used as input to classi ers based on Support Vector Machines to recognize upper facial action units and their all possible combinations. The system detects head gestures using Hidden Markov Models that use pupil positions in consecutive frames as observations. The system is evaluated on completely natural dataset with lots of head movements, pose changes and occlusions. The system can successfully detect head gestures 78.46% of time. Recognition accuracy of 67.83% for each individual AU is reported and the system can correctly identify all possible AU combinations with an accuracy of 61.25%. Thesis Supervisor: Rosalind W. Picard, Associate Professor of Media Arts and Sciences. This research was supported by NSF ROLE grant 0087768 and also is an output from a research project funded by Media Lab Asia. Media Lab Asia is funded in part by the Ministry of Information Technology, Government of India. The research was carried out in support of the Media Lab Asia Program by Massachusetts Institute of Technology's Media Laboratory itself. Media Lab Asia does not accept responsibility for any information provided or views expressed.